Some current volumetric capture systems (or virtual reality (VR) capture systems), such as Lytro Immerge, sparsely sample a light-field volume using a relatively large number of cameras. The cameras can be arranged, for example, over a lattice that covers a flat surface, or a 3D surface such as a sphere. The cameras may thus define a “tiled camera array.” The cameras may be traditional 2D cameras, array cameras, or plenoptic light-field cameras, as described in above-referenced U.S. Provisional Application 62/148,460. Whichever system is used, it is often difficult to avoid some sparseness of the samples, leaving gaps in the coverage. Thus, in order to provide virtual reality functionality, the rendering system may advantageously interpolate between camera views.
When a viewer is watching and interacting with the captured virtual reality environment (or “volume”), it is beneficial for the playback system to allow the viewer to have as many degrees of freedom as feasible to explore the captured volume. For, example, six degrees of freedom may be provided, so as to allow the user full angular viewing freedom (yaw, pitch, roll) as well as spatial viewing freedom (translation side-to-side, up and down, and forward and backward). In general, it is desirable to provide for movement along the degrees of freedom seamlessly and smoothly.
In general, the ability of a VR capture system to deliver high quality playback at interpolated viewpoints is limited by the density of the capture viewpoints and the ability of the system to interpolate between capture viewpoints, which may in turn depend on the quality of the system's estimation of various properties of objects in the world. In order to provide accurate interpolations, it is useful to have information about properties of surfaces and objects. Estimated world properties may include, for example and without limitation, 3D geometry, reflectance, specularity of materials, transparency/translucency of objects, and/or the like. In many situations, however, it may be difficult or impossible to estimate these world properties with a sufficient degree of accuracy or precision to provide the desired results. Fine objects like hair may also be problematic.
Various techniques can be used to improve the quality of the systems with respect to artifacts caused by errors in such estimated properties. One option may be to increase the density of cameras in the capture system, and thus reduce the amount of interpolation that is required. However, increasing density can increase system requirements, costs, storage, processing, heat management, and/or the like. Furthermore, there is a physical limit to how closely cameras may be spaced in the tiled camera array.
Other techniques for improving quality include improving the accuracy of the world estimation process. While such an approach may be appealing, it can be difficult or impossible within the constraints of the system. Another option may be to include additional types of sensors designed for specific purposes. Examples include LiDAR sensors, Time-of-Flight (ToF) sensors, and structured light. Specialized sensors may help improve the accuracy of certain aspects of the world estimation process, but may still not improve the level of accuracy to the desired level.